Problem Statement¶
Context¶
Businesses like banks which provide service have to worry about problem of 'Customer Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on improvement of service, keeping in mind these priorities.
Objective¶
You as a Data scientist with the bank need to build a neural network based classifier that can determine whether a customer will leave the bank or not in the next 6 months.
Data Dictionary¶
CustomerId: Unique ID which is assigned to each customer
Surname: Last name of the customer
CreditScore: It defines the credit history of the customer.
Geography: A customer’s location
Gender: It defines the Gender of the customer
Age: Age of the customer
Tenure: Number of years for which the customer has been with the bank
NumOfProducts: refers to the number of products that a customer has purchased through the bank.
Balance: Account balance
HasCrCard: It is a categorical variable which decides whether the customer has credit card or not.
EstimatedSalary: Estimated salary
isActiveMember: Is is a categorical variable which decides whether the customer is active member of the bank or not ( Active member in the sense, using bank products regularly, making transactions etc )
Exited : whether or not the customer left the bank within six month. It can take two values 0=No ( Customer did not leave the bank ) 1=Yes ( Customer left the bank )
Importing necessary libraries¶
In [1]:
# Installing the libraries with the specified version.
# !pip install tensorflow==2.15.0 scikit-learn==1.2.2 seaborn==0.13.1 matplotlib==3.7.1 numpy==1.25.2 pandas==2.0.3 imbalanced-learn==0.10.1 -q --user
In [2]:
# Libraries to help with reading and manipulating data
import pandas as pd
import numpy as np
import time
# libaries to help with data visualization
import matplotlib.pyplot as plt
import seaborn as sns
# Library to split data
from sklearn.model_selection import train_test_split
# library to import to standardize the data
from sklearn.preprocessing import StandardScaler, LabelEncoder
# importing different functions to build models
import tensorflow as tf
from tensorflow import keras
from keras import backend
from keras.models import Sequential
from keras.layers import Dense, Dropout
from tensorflow.keras.layers import BatchNormalization
# importing SMOTE
from imblearn.over_sampling import SMOTE
# importing metrics
from sklearn.metrics import confusion_matrix,roc_curve,classification_report,recall_score
import random
# Library to avoid the warnings
import warnings
warnings.filterwarnings("ignore")
Loading the dataset¶
In [3]:
from google.colab import drive
drive.mount('/content/drive')
In [4]:
churn = pd.read_csv('/content/drive/MyDrive/Python Course/Churn.csv')
# Make a copy of the data
ds = churn.copy()
Data Overview¶
View the first and last 5 rows of the dataset.¶
In [5]:
# let's view the first 5 rows of the data
ds.head()
Out[5]:
In [6]:
# let's view the last 5 rows of the data
ds.tail()
Out[6]:
Understand the shape of the dataset¶
In [7]:
# Checking the number of rows and columns in the training data
ds.shape
Out[7]:
- There are 10000 rows and 14 columns in the dataset.
Check the data types of the columns for the dataset¶
In [8]:
ds.info()
- There are 11 numeric columns and 3 object columns.
Checking the Statistical Summary¶
In [9]:
ds.describe()
Out[9]:
- The average credit score of customers is ~651 and the median is similar at ~652. 25% of customers have credit scores above 718.
- The average age of customers is ~39 years. The median age is 37 years. 25% of the customers have ages above 44 years. The max age is 92 years.
- The average custmoer tenure is ~5 years, which is similar to the median of 5.
- The average customer balance is ~76500. 25% of customers have a balance of 0.
- The average number of products of customers is ~1.5. The median is 1 and 25% of customers have ore than 2 products.
- Half of the customers have credit cards and are active members.
- The average and median estimated customer salary are ~100K. The min salary is about 12 dollars, which is probably a mistake. The max is ~200K dollars.
Checking for Missing Values¶
In [10]:
# let's check for missing values in the data
ds.isna().sum()
Out[10]:
- There are no null values.
Checking for Duplicate Values¶
In [11]:
ds.duplicated().sum()
Out[11]:
- There are no duplicate values.
Checking for unique values for each of the column¶
In [12]:
ds.nunique()
Out[12]:
- RowNumber and CustomerId are unique, hence should be dropped.
- Surname will not be useful also, so it should be dropped.
In [13]:
# Drop RowNumber, CustomerId and Surname
ds = ds.drop(['RowNumber', 'CustomerId', 'Surname'], axis=1)
- Preview some more of the data
In [14]:
for i in ds.describe(include=["object"]).columns:
print("Unique values in", i, "are :")
print(ds[i].value_counts())
print("*" * 50)
Exploratory Data Analysis¶
Univariate Analysis¶
In [15]:
# function to plot a boxplot and a histogram along the same scale.
def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
"""
Boxplot and histogram combined
data: dataframe
feature: dataframe column
figsize: size of figure (default (12,7))
kde: whether to show the density curve (default False)
bins: number of bins for histogram (default None)
"""
f2, (ax_box2, ax_hist2) = plt.subplots(
nrows=2, # Number of rows of the subplot grid= 2
sharex=True, # x-axis will be shared among all subplots
gridspec_kw={"height_ratios": (0.25, 0.75)},
figsize=figsize,
) # creating the 2 subplots
sns.boxplot(
data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
) # boxplot will be created and a star will indicate the mean value of the column
sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
) if bins else sns.histplot(
data=data, x=feature, kde=kde, ax=ax_hist2
) # For histogram
ax_hist2.axvline(
data[feature].mean(), color="green", linestyle="--"
) # Add mean to the histogram
ax_hist2.axvline(
data[feature].median(), color="black", linestyle="-"
) # Add median to the histogram
In [16]:
# function to create labeled barplots
def labeled_barplot(data, feature, perc=False, n=None):
"""
Barplot with percentage at the top
data: dataframe
feature: dataframe column
perc: whether to display percentages instead of count (default is False)
n: displays the top n category levels (default is None, i.e., display all levels)
"""
total = len(data[feature]) # length of the column
count = data[feature].nunique()
if n is None:
plt.figure(figsize=(count + 1, 5))
else:
plt.figure(figsize=(n + 1, 5))
plt.xticks(rotation=90, fontsize=15)
ax = sns.countplot(
data=data,
x=feature,
palette="Paired",
order=data[feature].value_counts().index[:n].sort_values(),
)
for p in ax.patches:
if perc == True:
label = "{:.1f}%".format(
100 * p.get_height() / total
) # percentage of each class of the category
else:
label = p.get_height() # count of each level of the category
x = p.get_x() + p.get_width() / 2 # width of the plot
y = p.get_height() # height of the plot
ax.annotate(
label,
(x, y),
ha="center",
va="center",
size=12,
xytext=(0, 5),
textcoords="offset points",
) # annotate the percentage
plt.show() # show the plot
Observations on CreditScore¶
In [17]:
histogram_boxplot(ds,'CreditScore', kde=True)
- The distribution of Credit Score for customers is mostly normal, with just a litte bit of left skewness.
- The mean and median are similar at about 651.
- There are outliers on the lower side of the Credit Score data.
Observations on Age¶
In [18]:
histogram_boxplot(ds, 'Age', kde=True)
- The distribution of Age of customers is right-skewed.
- The median and mean at 37 and about 39 years, respectively.
- The are outliers on the higher side of the data.
Observations on Balance¶
In [19]:
histogram_boxplot(ds, 'Balance', kde=True)
- 25% of the customers have a minimum balance of 0. Due to that fact, the data distribution initially, is to the left but for the rest of the data, the distribution of Balance is normal.
- The median and mean at ~97K and about 76K, respectively.
Observations on Estimated Salary¶
In [20]:
histogram_boxplot(ds, 'EstimatedSalary', kde=True)
- The distribution of Estimated Salary is uniform.
- The median and mean are similar at about 100K.
Observations on Exited¶
In [21]:
labeled_barplot(ds, "Exited", perc=True)
- ~80% of the customers did not exit, while ~20% exited.
Observations on Geography¶
In [22]:
labeled_barplot(ds, 'Geography', perc=True)
- Half of the customers are from France, while a quarter are from Germany and the remaining quarter from Spain.
Observations on Gender¶
In [23]:
labeled_barplot(ds, 'Gender', perc=True)
- ~45% of the customers are female, while ~55% are male.
Observations on Tenure¶
In [24]:
labeled_barplot(ds, 'Tenure', perc=True)
- Customers that have been with the company from 1 to 9 years make up majority of the customers at a combined 91%, with each of the individual year categories at between about 10-11%.
- Customers that have been with the company from 0 years are the least a ~4% and customers that hava a tenure of 10 years are at ~5%.
Observations on Number of Products¶
In [25]:
labeled_barplot(ds, 'NumOfProducts', perc=True)
- Customers that had only one product with the compan were the most at ~51%, followed by customers with two products at ~46%, customers with 3 products at ~3% and customers at below 1%.
Observations on Has Credit Card¶
In [26]:
labeled_barplot(ds, 'HasCrCard', perc=True)
- ~29% of customers have no credit cards while ~71% have credit cards.
Observations on Is Active Member¶
In [27]:
labeled_barplot(ds, 'IsActiveMember', perc=True)
- 48.5% of the customers are not active members, while 51.5% are active members.
Bivariate Analysis¶
In [28]:
# function to plot stacked bar chart
def stacked_barplot(data, predictor, target):
"""
Print the category counts and plot a stacked bar chart
data: dataframe
predictor: independent variable
target: target variable
"""
count = data[predictor].nunique()
sorter = data[target].value_counts().index[-1]
tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
by=sorter, ascending=False
)
print(tab1)
print("-" * 120)
tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
by=sorter, ascending=False
)
tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
plt.legend(
loc="lower left",
frameon=False,
)
plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
plt.show()
In [29]:
### Function to plot distributions
def distribution_plot_wrt_target(data, predictor, target):
fig, axs = plt.subplots(2, 2, figsize=(12, 10))
target_uniq = data[target].unique()
target_uniq.sort()
axs[0, 0].set_title("Distribution of target for target=" + str(target_uniq[0]))
sns.histplot(
data=data[data[target] == target_uniq[0]],
x=predictor,
kde=True,
ax=axs[0, 0],
color="teal",
)
axs[0, 1].set_title("Distribution of target for target=" + str(target_uniq[1]))
sns.histplot(
data=data[data[target] == target_uniq[1]],
x=predictor,
kde=True,
ax=axs[0, 1],
color="orange",
)
axs[1, 0].set_title("Boxplot w.r.t target")
sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")
axs[1, 1].set_title("Boxplot (without outliers) w.r.t target")
sns.boxplot(
data=data,
x=target,
y=predictor,
ax=axs[1, 1],
showfliers=False,
palette="gist_rainbow",
)
plt.tight_layout()
plt.show()
Correlation plot¶
In [30]:
# defining the list of numerical columns
cols_list = ["CreditScore","Age","Tenure","Balance","EstimatedSalary", "NumOfProducts"]
In [ ]:
# Pairplot with hue=Exited
sns.pairplot(ds, vars=cols_list, hue='Exited', diag_kind='kde')
#plt.show()
Out[ ]:
- Customers with ages between about 40 to about 70, exited more.
- Customers with 3-4 number of products, exited more.
In [ ]:
plt.figure(figsize=(15, 7))
sns.heatmap(ds[cols_list].corr(), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")
plt.show()
- There is a negative correlationship between Balance and NumOfProducts.
Exited Vs Geography¶
In [ ]:
stacked_barplot(ds, "Geography", "Exited" )
- The percentage of customers that exited was the most in Germany at ~32%. ~17% exited from Spain and ~16% from France.
Exited Vs Gender¶
In [ ]:
stacked_barplot(ds, "Gender", "Exited" )
- The percentage of female customers that exited is ~25% and ~16% for male customers. So, a higher percentage of females exited than males.
Exited Vs Has Credit Card¶
In [ ]:
stacked_barplot(ds, "HasCrCard", "Exited" )
- The percentage of customers that exited who had or did not have credit cards are similar at ~20 to 21%. So, this is not a factor in terms of determining exiting customers.
Exited Vs Is active member¶
In [ ]:
stacked_barplot(ds, "IsActiveMember", "Exited" )
- A higher percentage of non active members, ~27%, exited, than active members, ~14%.
Exited Vs Credit Score¶
In [ ]:
distribution_plot_wrt_target(ds, "CreditScore", "Exited")
- The distribution of Credit Score for customers who exited and those that did not is mostly normal, with just a slight of left skewness.
- The median credit score for the customers that did not exit is ~653, while for those that exited is ~646.
- There are some outliers on the lower credit score side for customers that exited.
Exited Vs Age¶
In [ ]:
distribution_plot_wrt_target(ds, "Age", "Exited")
- The Age distribution for customers that did not exit is left skewed but is normal for those hat exited.
- Those that exited have a higher median age at ~45 than those that did not exit at ~37.
- Both groups have outliers on the higher side of the age data.
Exited Vs Tenure¶
In [ ]:
plt.figure(figsize=(5,5))
sns.boxplot(y='Tenure',x='Exited',data=ds)
plt.show()
- The mean and median tenure of both customers that exited and those that did not are similar at 5.
Exited Vs Balance¶
In [ ]:
distribution_plot_wrt_target(ds, "Balance", "Exited")
- 25% of the customers that did not exit, have a minimum balance of 0. Due to that fact, the data distribution is initially to the left but for the rest of the data, the distribution of Balance is normal. The distribution is similar for the customers that exited.
- The median of the customers that did not exit is ~92K and ~109K for those that exited. So, customers with lower balances exited less than those with higher balances.
Exited Vs Number of Products¶
In [ ]:
plt.figure(figsize=(5,5))
sns.boxplot(y='NumOfProducts',x='Exited',data=ds)
plt.show()
- The median of customers that did not exit is 2, while for those that exited is 1 product.
- 25% of customers for both groups have 1 product, while 75% of the customers for both groups have 2 products.
- There are outliers in the data for customers who exited.
Exited Vs Estimated Salary¶
In [ ]:
distribution_plot_wrt_target(ds, "EstimatedSalary", "Exited")
- The distribution of Estimated Salary for both the customers that exited and those that did not exit, are similarly uniform.
- The median Estimated Salary for customers that did not exit is ~100K and ~102K for those that exited.
Data Preprocessing¶
Dummy Variable Creation¶
In [49]:
ds = pd.get_dummies(ds,columns=ds.select_dtypes(include=["object"]).columns.tolist(),drop_first=True)
ds = ds.astype(float)
ds.head()
Out[49]:
Train-validation-test Split¶
In [50]:
# set up the independent and dependent variables
X = ds.drop(['Exited'],axis=1)
y = ds['Exited'] # Exited
In [51]:
# Splitting the dataset into the Training and Testing set.
X_large, X_test, y_large, y_test = train_test_split(X, y, test_size = 0.15, random_state = 42,stratify=y,shuffle = True)
In [52]:
# Splitting the dataset into the Training and Validation set.
X_train, X_val, y_train, y_val = train_test_split(X_large, y_large, test_size = 0.17647, random_state = 42,stratify=y_large, shuffle = True)
- Split used: 70/15/15
In [53]:
print(X_train.shape, X_val.shape, X_test.shape)
In [54]:
print(y_train.shape, y_val.shape, y_test.shape)
Data Normalization¶
Since all the numerical values are on a different scales, bring all of them to the same scale by scaling.
In [55]:
# creating an instance of the standard scaler
sc = StandardScaler()
X_train[cols_list] = sc.fit_transform(X_train[cols_list])
X_val[cols_list] = sc.transform(X_val[cols_list])
X_test[cols_list] = sc.transform(X_test[cols_list])
Model Building¶
Model Evaluation Criterion¶
- The model can make wrong predictions by either predicting that a customer will exit and the customer doesn't exit or predicting that a customer will not exit and the customer exits.
- Of these two, the more important case is the latter, which is the model predicting that a customer will not exit but the customer exits, as this will amount to losing a valuable customer, potential business and revenue.
- These are the False Negatives. So, Recall, would be the metric to look at so that the bank can retain their valuable customers by identifying the customers who are at risk of exiting.
- The bank would want Recall to be maximized, as the greater the Recall, the higher the chances of minimizing false negatives. Hence, the focus should be on increasing Recall or minimizing the false negatives or in other words identifying the true positives(i.e. Exited = 1)
Functions & Set Ups¶
Let's create a function for plotting the confusion matrix
In [2113]:
def make_confusion_matrix(actual_targets, predicted_targets):
"""
To plot the confusion_matrix with percentages
actual_targets: actual target (dependent) variable values
predicted_targets: predicted target (dependent) variable values
"""
cm = confusion_matrix(actual_targets, predicted_targets)
labels = np.asarray(
[
["{0:0.0f}".format(item) + "\n{0:.2%}".format(item / cm.flatten().sum())]
for item in cm.flatten()
]
).reshape(cm.shape[0], cm.shape[1])
plt.figure(figsize=(6, 4))
sns.heatmap(cm, annot=labels, fmt="")
plt.ylabel("True label")
plt.xlabel("Predicted label")
Create the plot function for repetitive use
In [2114]:
def plot(history, name):
"""
Function to plot loss/accuracy
history: an object which stores the metrics and losses.
name: can be one of Loss or Accuracy
"""
fig, ax = plt.subplots() #Creating a subplot with figure and axes.
plt.plot(history.history[name]) #Plotting the train accuracy or train loss
plt.plot(history.history['val_'+name]) #Plotting the validation accuracy or validation loss
plt.title('Model ' + name.capitalize()) #Defining the title of the plot.
plt.ylabel(name.capitalize()) #Capitalizing the first letter.
plt.xlabel('Epoch') #Defining the label for the x-axis.
fig.legend(['Train', 'Validation'], loc="outside right upper") #Defining the legend, loc controls the position of the legend.
Let's create two blank dataframes that will store the recall values for all the models we build.
In [2115]:
train_metric_df = pd.DataFrame(columns=["recall"])
valid_metric_df = pd.DataFrame(columns=["recall"])
Create a more comprehensive results dataframe
In [2116]:
columns = ["# hidden layers","# neurons - hidden layer","activation function - hidden layer ","# epochs","batch size","optimizer","time(secs)","Train_loss","Valid_loss","Train_Recall","Valid_Recall"]
results = pd.DataFrame(columns=columns)
Neural Network with SGD Optimizer (NO HIDDEN LAYER)¶
- Let's start with this as a baseline model - no hidden layers
In [2117]:
# Clear the session memory
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(2)
random.seed(2)
tf.random.set_seed(2)
In [2118]:
# Initializing the neural network
model_nhl = Sequential()
# Adding the input layer with neurons
model_nhl.add(Dense(1, activation = 'sigmoid', input_dim = X_train.shape[1]))
In [2119]:
# Use SGD as the optimizer.
optimizer = tf.keras.optimizers.SGD(0.001)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
In [2120]:
## Complete the code to compile the model with binary cross entropy as loss function and recall as the metric.
model_nhl.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
In [2121]:
model_nhl.summary()
In [2122]:
epochs = 50
batch_size = X_train.shape[0]
In [2123]:
# Get start time
start = time.time()
# Fitting the ANN
history_nhl = model_nhl.fit(
X_train, y_train,
batch_size=batch_size,
validation_data=(X_val,y_val),
epochs=epochs,
verbose=1
)
# Get end time
end=time.time()
In [2124]:
# Time taken
print("Time taken in seconds ",end-start)
In [2125]:
model_nhl.evaluate(X_train,y_train)
Out[2125]:
In [2126]:
model_nhl.evaluate(X_val,y_val)
Out[2126]:
Loss function
In [2127]:
# Plotting Train Loss vs Validation Loss
plot(history_nhl, 'loss')
Recall
In [2128]:
# Plotting Train recall vs Validation recall
plot(history_nhl, 'recall')
In [2129]:
# Predicting the results using best as a threshold
y_train_pred = model_nhl.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
Out[2129]:
In [2130]:
# Predicting the results using best as a threshold
y_val_pred = model_nhl.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
Out[2130]:
In [2131]:
model_name = "NN with SGD with No Hidden Layers"
train_metric_df.loc[model_name] = recall_score(y_train, y_train_pred)
valid_metric_df.loc[model_name] = recall_score(y_val, y_val_pred)
In [2132]:
results.loc[model_name]=['-','-','-',epochs,batch_size,'SGD',(end-start),history_nhl.history["loss"][-1],history_nhl.history["val_loss"][-1],history_nhl.history["recall"][-1],history_nhl.history["val_recall"][-1]]
In [2133]:
results
Out[2133]:
Classification report
In [2134]:
# Classification report
cr = classification_report(y_train, y_train_pred)
print(cr)
In [2135]:
# Classification report
cr = classification_report(y_val, y_val_pred)
print(cr)
Confusion matrix
In [2136]:
make_confusion_matrix(y_train, y_train_pred)
In [2137]:
make_confusion_matrix(y_val, y_val_pred)
- This Recall score is not very high. The losses seem to be high. Let's proceed to improve these.
Model Performance Improvement¶
Neural Network with SGD Optimizer¶
- 1 Hidden Layer [14 neurons]
In [2138]:
# Clear the session memory
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(2)
random.seed(2)
tf.random.set_seed(2)
In [2139]:
# Initializing the neural network
model_0 = Sequential()
# Adding the input layer with neurons and relu as the activation function
model_0.add(Dense(14, activation='relu', input_dim = X_train.shape[1]))
# Adding the output layer with 1 neuron for binary output and using sigmoid for binary output
model_0.add(Dense(1, activation = 'sigmoid'))
In [2140]:
# Use SGD as the optimizer.
optimizer = tf.keras.optimizers.SGD(0.001)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
In [2141]:
## Complete the code to compile the model with binary cross entropy as loss function and recall as the metric.
model_0.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
In [2142]:
model_0.summary()
In [2143]:
epochs = 50
batch_size = X_train.shape[0]
In [2144]:
# Get start time
start = time.time()
# Fitting the ANN
history_0 = model_0.fit(
X_train, y_train,
batch_size=batch_size,
validation_data=(X_val,y_val),
epochs=epochs,
verbose=1
)
# Get end time
end=time.time()
In [2145]:
# Time taken
print("Time taken in seconds ",end-start)
In [2146]:
model_0.evaluate(X_train,y_train)
Out[2146]:
In [2147]:
model_0.evaluate(X_val,y_val)
Out[2147]:
Loss function
In [2148]:
# Plotting Train Loss vs Validation Loss
plot(history_0, 'loss')
Recall
In [2149]:
# Plotting Train recall vs Validation recall
plot(history_0, 'recall')
In [2150]:
# Predicting the results using best as a threshold
y_train_pred = model_0.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
Out[2150]:
In [2151]:
# Predicting the results using best as a threshold
y_val_pred = model_0.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
Out[2151]:
In [2152]:
model_name = "NN with SGD - 1 Hidden Layer"
train_metric_df.loc[model_name] = recall_score(y_train, y_train_pred)
valid_metric_df.loc[model_name] = recall_score(y_val, y_val_pred)
In [2153]:
results.loc[model_name]=['1','14','relu',epochs,batch_size,'SGD',(end-start),history_0.history["loss"][-1],history_0.history["val_loss"][-1],history_0.history["recall"][-1],history_0.history["val_recall"][-1]]
In [2154]:
results
Out[2154]:
Classification report
In [2155]:
# Classification report
cr = classification_report(y_train, y_train_pred)
print(cr)
In [2156]:
# Classification report
cr = classification_report(y_val, y_val_pred)
print(cr)
Confusion matrix
In [2157]:
make_confusion_matrix(y_train, y_train_pred)
In [2158]:
make_confusion_matrix(y_val, y_val_pred)
- This Recall score is not bad at all and the loss improved.
- The addition of one hidden layer to the baseline model (no hidden layers), brought about a dramatic improvement in the Recall score. The loss also reduced quite a bit.
- I noticed that the Recall goes down, though, as the epochs increase. Let's try to improve on this.
Neural Network with SGD Optimizer¶
- 2 Hidden Layers [14,7 neurons]
In [2159]:
# Clear the session memory
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(2)
random.seed(2)
tf.random.set_seed(2)
In [2160]:
# Initializing the neural network
model_01 = Sequential()
# Adding the input layer with neurons and relu as the activation function
model_01.add(Dense(14, activation='relu', input_dim = X_train.shape[1]))
# Adding one hidden layer with neurons and Relu as the activation function
model_01.add(Dense(7, activation='relu'))
# Adding the output layer with 1 neuron for binary output and using sigmoid for binary output
model_01.add(Dense(1, activation = 'sigmoid'))
In [2161]:
# Use SGD as the optimizer.
optimizer = tf.keras.optimizers.SGD(0.001)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
In [2162]:
## Complete the code to compile the model with binary cross entropy as loss function and recall as the metric.
model_01.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
In [2163]:
model_01.summary()
In [2164]:
epochs = 50
batch_size = X_train.shape[0]
In [2165]:
# Get start time
start = time.time()
# Fitting the ANN
history_01 = model_01.fit(
X_train, y_train,
batch_size=batch_size,
validation_data=(X_val,y_val),
epochs=epochs,
verbose=1
)
# Get end time
end=time.time()
In [2166]:
# Time taken
print("Time taken in seconds ",end-start)
In [2167]:
model_01.evaluate(X_train,y_train)
Out[2167]:
In [2168]:
model_01.evaluate(X_val,y_val)
Out[2168]:
Loss function
In [2169]:
# Plotting Train Loss vs Validation Loss
plot(history_01, 'loss')
Recall
In [2170]:
# Plotting Train recall vs Validation recall
plot(history_01, 'recall')
In [2171]:
# Predicting the results using best as a threshold
y_train_pred = model_01.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
Out[2171]:
In [2172]:
# Predicting the results using best as a threshold
y_val_pred = model_01.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
Out[2172]:
In [2173]:
model_name = "NN with SGD - 2 Hidden Layers [14,7]"
train_metric_df.loc[model_name] = recall_score(y_train, y_train_pred)
valid_metric_df.loc[model_name] = recall_score(y_val, y_val_pred)
In [2174]:
results.loc[model_name]=['2','14,7','relu,relu',epochs,batch_size,'SGD',(end-start),history_01.history["loss"][-1],history_01.history["val_loss"][-1],history_01.history["recall"][-1],history_01.history["val_recall"][-1]]
In [2175]:
results
Out[2175]:
Classification report
In [2176]:
# Classification report
cr = classification_report(y_train, y_train_pred)
print(cr)
In [2177]:
# Classification report
cr = classification_report(y_val, y_val_pred)
print(cr)
Confusion matrix
In [2178]:
make_confusion_matrix(y_train, y_train_pred)
In [2179]:
make_confusion_matrix(y_val, y_val_pred)
- This Recall score is very much improved and the loss improved also.
- The addition of a secong hidden layer to the baseline model resulted in even more performance improvement as both the Recall score increased and the loss reduced further.
- I noticed that the Recall still goes down, though, as the epochs increase. Let's try to improve on this.
Neural Network with SGD Optimizer¶
- 2 Hidden Layers [64,32 neurons]
In [2180]:
# Clear the session memory
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(2)
random.seed(2)
tf.random.set_seed(2)
In [2181]:
# Initializing the neural network
model_02 = Sequential()
# Adding the input layer with neurons and relu as the activation function
model_02.add(Dense(64, activation='relu', input_dim = X_train.shape[1]))
# Adding one hidden layer with neurons and Relu as the activation function
model_02.add(Dense(32, activation='relu'))
# Adding the output layer with 1 neuron for binary output and using sigmoid for binary output
model_02.add(Dense(1, activation = 'sigmoid'))
In [2182]:
# Use SGD as the optimizer.
optimizer = tf.keras.optimizers.SGD(0.001)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
In [2183]:
## Complete the code to compile the model with binary cross entropy as loss function and recall as the metric.
model_02.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
In [2184]:
model_02.summary()
In [2185]:
epochs = 50
batch_size = X_train.shape[0]
In [2186]:
# Get start time
start = time.time()
# Fitting the ANN
history_02 = model_02.fit(
X_train, y_train,
batch_size=batch_size,
validation_data=(X_val,y_val),
epochs=epochs,
verbose=1
)
# Get end time
end=time.time()
In [2187]:
# Time taken
print("Time taken in seconds ",end-start)
In [2188]:
model_02.evaluate(X_train,y_train)
Out[2188]:
In [2189]:
model_02.evaluate(X_val,y_val)
Out[2189]:
Loss function
In [2190]:
# Plotting Train Loss vs Validation Loss
plot(history_02, 'loss')
Recall
In [2191]:
# Plotting Train recall vs Validation recall
plot(history_02, 'recall')
In [2192]:
# Predicting the results using best as a threshold
y_train_pred = model_02.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
Out[2192]:
In [2193]:
# Predicting the results using best as a threshold
y_val_pred = model_02.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
Out[2193]:
In [2194]:
model_name = "NN with SGD - 2 Hidden Layers [64,32]"
train_metric_df.loc[model_name] = recall_score(y_train, y_train_pred)
valid_metric_df.loc[model_name] = recall_score(y_val, y_val_pred)
In [2195]:
results.loc[model_name]=['2','64,32','relu,relu',epochs,batch_size,'SGD',(end-start),history_02.history["loss"][-1],history_02.history["val_loss"][-1],history_02.history["recall"][-1],history_02.history["val_recall"][-1]]
In [2196]:
results
Out[2196]:
Classification report
In [2197]:
# Classification report
cr = classification_report(y_train, y_train_pred)
print(cr)
In [2198]:
# Classification report
cr = classification_report(y_val, y_val_pred)
print(cr)
Confusion matrix
In [2199]:
make_confusion_matrix(y_train, y_train_pred)
In [2200]:
make_confusion_matrix(y_val, y_val_pred)
- Increasing the number of neurons reduced the Recall drastically. It also reduced the loss. It pretty much drained the life out of the base model.
Neural Network with Adam Optimizer¶
- 2 Hidden Layers [64,32 neurons]
In [2201]:
# Clear the session memory
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(2)
random.seed(2)
tf.random.set_seed(2)
In [2202]:
# Initializing the neural network
model_1 = Sequential()
model_1.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
model_1.add(Dense(32,activation='relu'))
model_1.add(Dense(1, activation = 'sigmoid'))
In [2203]:
# Complete the code to use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
#metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
In [2204]:
# Complete the code to compile the model with binary cross entropy as loss function and recall as the metric
model_1.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
In [2205]:
model_1.summary()
In [2206]:
epochs = 50
batch_size = 128
In [2207]:
# Get start time
start = time.time()
# Fitting the ANN
history_1 = model_1.fit(
X_train,y_train,
batch_size=batch_size,
validation_data=(X_val,y_val),
epochs=epochs,
verbose=1
)
# Get end time
end=time.time()
In [2208]:
# Time taken
print("Time taken in seconds ",end-start)
In [2209]:
model_1.evaluate(X_train,y_train)
Out[2209]:
In [2210]:
model_1.evaluate(X_val,y_val)
Out[2210]:
Loss function
In [2211]:
# Plotting Train Loss vs Validation Loss
plot(history_1, 'loss')
Recall
In [2212]:
# Plotting Train recall vs Validation recall
plot(history_1, 'recall')
In [2213]:
#Predicting the results using 0.5 as the threshold
y_train_pred = model_1.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
Out[2213]:
In [2214]:
#Predicting the results using 0.5 as the threshold
y_val_pred = model_1.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
Out[2214]:
In [2215]:
model_name = "NN with Adam - 2 Hidden Layers [64,32]"
train_metric_df.loc[model_name] = recall_score(y_train,y_train_pred)
valid_metric_df.loc[model_name] = recall_score(y_val,y_val_pred)
In [2216]:
results.loc[model_name]=['2','64,32','relu,relu',epochs,batch_size,'Adam',(end-start),history_1.history["loss"][-1],history_1.history["val_loss"][-1],history_1.history["recall"][-1],history_1.history["val_recall"][-1]]
In [2217]:
results
Out[2217]:
Classification report
In [2218]:
#lassification report
cr=classification_report(y_train,y_train_pred)
print(cr)
In [2219]:
#classification report
cr=classification_report(y_val, y_val_pred)
print(cr)
Confusion matrix
In [2220]:
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
In [2221]:
#Calculating the confusion matrix
make_confusion_matrix(y_val, y_val_pred)
- The Recall on this model improved over the last model and the loss reduced further. Nothing significant versus the last model.
In [2222]:
results
Out[2222]:
Neural Network with Adam Optimizer and Dropout¶
- 2 Hidden Layers [64,32 neurons]
In [2328]:
# Clear up the session memory
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(2)
random.seed(2)
tf.random.set_seed(2)
In [2329]:
# Initializing the neural network
model_2 = Sequential()
# Add the input layer with neurons and relu as activation function
model_2.add(Dense(64,activation='relu',input_dim = X_train.shape[1]))
# Add dropout with ratio of 0.2 or any suitable value.
model_2.add(Dropout(0.2))
# Add a hidden layer
model_2.add(Dense(32,activation='tanh'))
# Add dropout with ratio of 0.1 or any suitable value.
model_2.add(Dropout(0.2))
# Add the number of neurons required in the output layer.
model_2.add(Dense(1, activation = 'sigmoid'))
In [2330]:
# Complete the code to use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
In [2331]:
## Complete the code to compile the model with binary cross entropy as loss function and recall as the metric.
model_2.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
In [2332]:
# Summary of the model
model_2.summary()
In [2333]:
epochs = 50
batch_size = 128
In [2334]:
# Get start time
start = time.time()
# Fitting the ANN with batch_size = 32 and 100 epochs
history_2 = model_2.fit(
X_train,y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(X_val,y_val)
)
# Get end time
end=time.time()
In [2335]:
# Time taken
print("Time taken in seconds ",end-start)
Loss function
In [2336]:
# Plotting Train Loss vs Validation Loss
plot(history_2, 'loss')
In [2337]:
# Plotting Train recall vs Validation recall
plot(history_2, 'recall')
In [2338]:
#Predicting the results using best as a threshold
y_train_pred = model_2.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
Out[2338]:
In [2339]:
#Predicting the results using 0.5 as the threshold.
y_val_pred = model_2.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
Out[2339]:
In [2340]:
model_name = "NN with Adam & Dropout - 2 Hidden Layers [64,32]"
train_metric_df.loc[model_name] = recall_score(y_train,y_train_pred)
valid_metric_df.loc[model_name] = recall_score(y_val,y_val_pred)
In [2341]:
results.loc[model_name]=['2','64,32,','relu,tanh',epochs,batch_size,'Adam',(end-start),history_2.history["loss"][-1],history_2.history["val_loss"][-1],history_2.history["recall"][-1],history_2.history["val_recall"][-1]]
In [2347]:
results
Out[2347]:
Classification report
In [2343]:
#classification report
cr=classification_report(y_train,y_train_pred)
print(cr)
In [2344]:
#classification report
cr = classification_report(y_val,y_val_pred)
print(cr)
Confusion matrix
In [2345]:
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
In [2346]:
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
- The addition of the Dropout parameter dis not seem to make a difference as the results are similar.
Neural Network with Adam Optimizer and Dropout¶
- 4 Hidden Layers [32,32,32,32 neurons]
- Added to see how a 4 Hidden-Layers model will do
In [2223]:
# Clear up the session memory
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(2)
random.seed(2)
tf.random.set_seed(2)
In [2224]:
# Initializing the neural network
model_21 = Sequential()
# Add the input layer with neurons and relu as activation function
model_21.add(Dense(32,activation='relu',input_dim = X_train.shape[1]))
# Add dropout with ratio of 0.2 or any suitable value.
model_21.add(Dropout(0.2))
# Add a hidden layer
model_21.add(Dense(32,activation='tanh'))
# Add a hidden layer
model_21.add(Dense(32,activation='relu'))
# Add dropout with ratio of 0.1 or any suitable value.
model_21.add(Dropout(0.1))
# Add a hidden layer
model_21.add(Dense(32,activation='relu'))
# Add the number of neurons required in the output layer.
model_21.add(Dense(1, activation = 'sigmoid'))
In [2225]:
# Complete the code to use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
In [2226]:
## Complete the code to compile the model with binary cross entropy as loss function and recall as the metric.
model_21.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
In [2227]:
# Summary of the model
model_21.summary()
In [2228]:
epochs = 50
batch_size = 128
In [2229]:
# Get start time
start = time.time()
# Fitting the ANN with batch_size = 32 and 100 epochs
history_21 = model_21.fit(
X_train,y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(X_val,y_val)
)
# Get end time
end=time.time()
In [2230]:
# Time taken
print("Time taken in seconds ",end-start)
Loss function
In [2231]:
# Plotting Train Loss vs Validation Loss
plot(history_21, 'loss')
In [2232]:
# Plotting Train recall vs Validation recall
plot(history_21, 'recall')
In [2233]:
#Predicting the results using best as a threshold
y_train_pred = model_21.predict(X_train)
y_train_pred = (y_train_pred > 0.5)
y_train_pred
Out[2233]:
In [2234]:
#Predicting the results using 0.5 as the threshold.
y_val_pred = model_21.predict(X_val)
y_val_pred = (y_val_pred > 0.5)
y_val_pred
Out[2234]:
In [2235]:
model_name = "NN with Adam & Dropout - 4 Hidden Layers [32,32,32,32]"
train_metric_df.loc[model_name] = recall_score(y_train,y_train_pred)
valid_metric_df.loc[model_name] = recall_score(y_val,y_val_pred)
In [2236]:
results.loc[model_name]=['4','32,32,32,32','relu,tanh,relu,relu',epochs,batch_size,'Adam',(end-start),history_21.history["loss"][-1],history_21.history["val_loss"][-1],history_21.history["recall"][-1],history_21.history["val_recall"][-1]]
In [2237]:
results
Out[2237]:
Classification report
In [2238]:
#classification report
cr=classification_report(y_train,y_train_pred)
print(cr)
In [2239]:
#classification report
cr = classification_report(y_val,y_val_pred)
print(cr)
Confusion matrix
In [2240]:
#Calculating the confusion matrix
make_confusion_matrix(y_train, y_train_pred)
In [2241]:
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
- This model performed a tad worse than the previous model and the loss is similar.
- This model performed similar to the previous one. There is not much improvement.
Neural Network with Balanced Data (by applying SMOTE) and SGD Optimizer¶
- 2 Hidden Layers [14,7 neurons]
Let's try to apply SMOTE to balance this dataset and then again apply hyperparamter tuning accordingly.
In [2261]:
sm = SMOTE(random_state=42)
# Complete the code to fit SMOTE on the training data.
X_train_smote, y_train_smote= sm.fit_resample(X_train, y_train)
print('After UpSampling, the shape of train_X: {}'.format(X_train_smote.shape))
print('After UpSampling, the shape of train_y: {} \n'.format(y_train_smote.shape))
In [2262]:
y_train_smote.value_counts()
Out[2262]:
Let's build a model with the balanced dataset
In [2263]:
# Clear the session memory
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(2)
random.seed(2)
tf.random.set_seed(2)
In [2264]:
# Initializing the model
model_3 = Sequential()
# Add a input layer
model_3.add(Dense(14,activation='relu',input_dim = X_train_smote.shape[1]))
# Add a hidden layer
model_3.add(Dense(7,activation='relu'))
# Add the required number of neurons in the output layer with a sigmoid activation function.
model_3.add(Dense(1, activation = 'sigmoid'))
In [2265]:
#Complete the code to use SGD as the optimizer.
optimizer = tf.keras.optimizers.SGD(0.001)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
In [2266]:
# Complete the code to compile the model with binary cross entropy as loss function and recall as the metric
model_3.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
In [2267]:
model_3.summary()
In [2268]:
epochs = 25
batch_size = 128
In [2269]:
# Get start time
start = time.time()
# Fitting the ANN
history_3 = model_3.fit(
X_train_smote, y_train_smote,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val)
)
# Get end time
end=time.time()
In [2270]:
# Time taken
print("Time taken in seconds ",end-start)
In [2271]:
model_3.evaluate(X_train_smote,y_train_smote)
Out[2271]:
In [2272]:
model_3.evaluate(X_val,y_val)
Out[2272]:
Loss function
In [2273]:
#Plotting Train Loss vs Validation Loss
plot(history_3, 'loss')
In [2274]:
# Plotting Train recall vs Validation recall
plot(history_3, 'recall')
In [2275]:
y_train_pred = model_3.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
Out[2275]:
In [2276]:
y_val_pred = model_3.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
Out[2276]:
In [2277]:
model_name = "NN with SMOTE & SGD - 2 Hidden Layers [14,7]"
train_metric_df.loc[model_name] = recall_score(y_train_smote,y_train_pred)
valid_metric_df.loc[model_name] = recall_score(y_val,y_val_pred)
In [2278]:
results.loc[model_name]=['2','14,7','relu,relu',epochs,batch_size,'Adam',(end-start),history_3.history["loss"][-1],history_3.history["val_loss"][-1],history_3.history["recall"][-1],history_3.history["val_recall"][-1]]
In [2279]:
results
Out[2279]:
Classification report
In [2280]:
cr=classification_report(y_train_smote,y_train_pred)
print(cr)
In [2281]:
cr=classification_report(y_val,y_val_pred)
print(cr)
Confusion matrix
In [2282]:
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
In [2283]:
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
- This model with balancing the data didn't do badly. The Recall score improved from the last few model but the loss increased.
- The data balancing helped to improve the performance.
Neural Network with Balanced Data (by applying SMOTE) and Adam Optimizer¶
- 3 Hidden Layers [32,32,32 neurons]
Let's build a model with the balanced dataset
In [2284]:
# Clear the session memmory
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(2)
random.seed(2)
tf.random.set_seed(2)
In [2285]:
# Initializing the model
model_4 = Sequential()
# Add a input layer
model_4.add(Dense(32,activation='relu',input_dim = X_train_smote.shape[1]))
# Add a hidden layer
model_4.add(Dense(32,activation='relu'))
#Add a hidden layer
model_4.add(Dense(32,activation='relu'))
# Add the required number of neurons in the output layer and a suitable activation function.
model_4.add(Dense(1, activation = 'sigmoid'))
In [2286]:
model_4.summary()
In [2287]:
#Complete the code to use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam()
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
In [2288]:
# Complete the code to compile the model with binary cross entropy as loss function and recall as the metric
model_4.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
In [2289]:
model_4.summary()
In [2290]:
epochs = 25
batch_size = 128
In [2291]:
# Get start time
start = time.time()
# Fitting the ANN
history_4 = model_4.fit(
X_train_smote,y_train_smote,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val)
)
# Get end time
end=time.time()
In [2292]:
# Time taken
print("Time taken in seconds ",end-start)
In [2293]:
model_4.evaluate(X_train_smote,y_train_smote)
Out[2293]:
In [2294]:
model_4.evaluate(X_val,y_val)
Out[2294]:
Loss function
In [2295]:
# Plotting Train Loss vs Validation Loss
plot(history_4, 'loss')
In [2296]:
# Plotting Train recall vs Validation recall
plot(history_4, 'recall')
In [2297]:
y_train_pred = model_4.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
Out[2297]:
In [2298]:
y_val_pred = model_4.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
Out[2298]:
In [2299]:
model_name = "NN with SMOTE & Adam - 3 Hidden Layers [32,32,32]"
train_metric_df.loc[model_name] = recall_score(y_train_smote,y_train_pred)
valid_metric_df.loc[model_name] = recall_score(y_val,y_val_pred)
In [2300]:
results.loc[model_name]=['3','32,32,32','relu,relu,relu',epochs,batch_size,'Adam',(end-start),history_4.history["loss"][-1],history_4.history["val_loss"][-1],history_4.history["recall"][-1],history_3.history["val_recall"][-1]]
In [2301]:
results
Out[2301]:
Classification report
In [2302]:
cr=classification_report(y_train_smote,y_train_pred)
print(cr)
In [2303]:
cr=classification_report(y_val, y_val_pred)
print(cr)
Confusion matrix
In [2304]:
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
In [2305]:
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred)
- Again, it looks like the balancing of the data is having a positive effect. This model performed better than the previous balanced data model. It improved both in the Recall score and loss.
Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout¶
- 3 Hidden Layers [32,32,32 neurons]
In [2306]:
# Clear the session memory
backend.clear_session()
#Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(2)
random.seed(2)
tf.random.set_seed(2)
In [2307]:
# Initializing the model
model_5 = Sequential()
# Add required # of neurons to the input layer with relu as activation function
model_5.add(Dense(32,activation='relu',input_dim = X_train_smote.shape[1]))
#model_5.add(BatchNormalization())
# Add dropout rate
model_5.add(Dropout(0.2))
# Add required # neurons to the hidden layer with any activation function.
model_5.add(Dense(32,activation='relu'))
#model_5.add(BatchNormalization())
# Add dropout rate.
model_5.add(Dropout(0.3))
# Add hidden layer with neurons with relu as activation function
model_5.add(Dense(32,activation='relu'))
# Add the required number of neurons in the output layer with a suitable activation function.
model_5.add(Dense(1, activation = 'sigmoid'))
In [2308]:
# Complete the code to use Adam as the optimizer.
optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)
# uncomment one of the following lines to define the metric to be used
# metric = 'accuracy'
metric = keras.metrics.Recall()
# metric = keras.metrics.Precision()
# metric = keras.metrics.F1Score()
In [2309]:
# Complete the code to compile the model with binary cross entropy as loss function and recall as the metric
model_5.compile(loss='binary_crossentropy',optimizer=optimizer,metrics=[metric])
In [2310]:
model_5.summary()
In [2311]:
epochs = 25
batch_size = 128
In [2312]:
# Get start time
start = time.time()
history_5 = model_5.fit(
X_train_smote,y_train_smote,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data = (X_val,y_val))
# Get end time
end=time.time()
In [2313]:
# Time taken
print("Time taken in seconds ",end-start)
In [2314]:
model_5.evaluate(X_train_smote,y_train_smote)
Out[2314]:
In [2315]:
model_5.evaluate(X_val,y_val)
Out[2315]:
Loss function
In [2316]:
# Plotting Train Loss vs Validation Loss
plot(history_5, 'loss')
In [2317]:
# Plotting Train recall vs Validation recall
plot(history_5, 'recall')
In [2318]:
y_train_pred = model_5.predict(X_train_smote)
#Predicting the results using 0.5 as the threshold
y_train_pred = (y_train_pred > 0.5)
y_train_pred
Out[2318]:
In [2319]:
y_val_pred = model_5.predict(X_val)
#Predicting the results using 0.5 as the threshold
y_val_pred = (y_val_pred > 0.5)
y_val_pred
Out[2319]:
In [2320]:
model_name = "NN with SMOTE,Adam & Dropout - 3 Hidden Layers [32,32,32]"
train_metric_df.loc[model_name] = recall_score(y_train_smote,y_train_pred)
valid_metric_df.loc[model_name] = recall_score(y_val,y_val_pred)
In [2321]:
results.loc[model_name]=['3','32,32,32','relu,relu,relu',epochs,batch_size,'Adam',(end-start),history_5.history["loss"][-1],history_5.history["val_loss"][-1],history_5.history["recall"][-1],history_5.history["val_recall"][-1]]
In [2322]:
results
Out[2322]:
Classification report
In [2323]:
cr=classification_report(y_train_smote,y_train_pred)
print(cr)
In [2324]:
#classification report
cr=classification_report(y_val, y_val_pred) ## Complete the code to check the model's performance on the validation set
print(cr)
Confusion matrix
In [2325]:
#Calculating the confusion matrix
make_confusion_matrix(y_train_smote, y_train_pred)
In [2326]:
#Calculating the confusion matrix
make_confusion_matrix(y_val,y_val_pred) ## Complete the code to check the model's performance on the validation set
- The addition of the Dropout parameter did not improve the previous model. The Recall is lower while the loss is higher.
Model Performance Comparison and Final Model Selection¶
In [2353]:
results
Out[2353]:
- Calculate the difference is the losses for each model
In [2351]:
results.Train_loss - results.Valid_loss
Out[2351]:
- Among all models, the NN with SGD - 2 Hidden Layers [14,7] model achieved the highest training and validation scores.
- A training Recall score of 0.945302 and a validation Recall score of 0.963934 suggests that the model is performing slightly better on the validation data compared to the training data.
- This indicates that the model is generalizing well to unseen data.
- We will use this model as our final model.
In [2354]:
y_test_pred = model_01.predict(X_test)
y_test_pred = (y_test_pred > 0.5)
print(y_test_pred)
In [2355]:
#lets print classification report
cr=classification_report(y_test,y_test_pred)
print(cr)
In [2356]:
#Calculating the confusion matrix
make_confusion_matrix(y_test,y_test_pred)
- The model generalized very well on unseen data, as the test data Recall score is .95
Actionable Insights and Business Recommendations¶
Our neural network model has a Recall score of .95, so it should do well with predicting customers that will indeed exit.
Our analyses revealed that:
- the percentage of customers that exited was the most in Germany; so the bank can try to run some marketing campaigns geared towards the German customers.
- a higher percentage of females exited than males; so the bank can try to run some appropriate marketing campaigns geared towards getting more female customers to not exit.
- a higher percentage of inactive members, exited, than active members; so the bank can come up with initiatives to get members active by giving incentives to inactive members to use their products like gift cards or discounts.
- customers with ages between about 40 to about 70, exited more and with a higher median age of about 45 than customers that did not exit, at about 37; so the bank can target this age group to ensure that they retain them as customers.
- customers with 3-4 number of products, exited more; so the bank can come up with ways to reduce the number of products customers use but make sure that they are actively using this reduced number of products.
Power Ahead